Near Real-Time CloudTrail Tampering Detection and Automated Remediation

The Challenge

AWS CloudTrail is the foundation of your AWS audit trail. It records every API call made in your account and delivers those records to Amazon Simple Storage Service (Amazon S3) and optionally to Amazon CloudWatch Logs. Protecting the integrity of that trail is critical: if logging is stopped or the trail is deleted, you lose continuous visibility into account activity, which is essential for your security operations.

Without near real-time detection, an unauthorized actor who disables CloudTrail logging has a window to operate undetected. AWS Config can detect the configuration drift, but its evaluation cycle may not run immediately. During that gap, the unauthorized actor can perform unauthorized actions with no audit record. This solution closes that gap by detecting unauthorized configuration changes as soon as they occur and optionally re-enabling logging automatically.

This solution deploys a layered detection and response stack that alerts your security team shortly after a CloudTrail unauthorized configuration change occurs and, optionally, automatically re-enables logging without human intervention. It uses CloudWatch metric filters and CloudWatch alarms for durable log-based detection, Amazon EventBridge rules for near real-time event matching, Amazon Simple Notification Service (Amazon SNS) for alert delivery, and an optional AWS Lambda function for automated remediation.

Prerequisites

Before deploying this solution, verify the following are in place:

• AWS CloudTrail trail - An active trail in the target region, configured to deliver management events.
• CloudWatch Logs integration - The trail must be configured to deliver logs to a CloudWatch Logs log group. Note the log group name, as you need it as a parameter.
• Amazon SNS email confirmation - After stack creation, Amazon SNS sends a subscription confirmation email to the address you provide. You must click the confirmation link. Alerts are not delivered until the subscription is confirmed.
• AWS Identity and Access Management (IAM) permissions - The IAM principal deploying the stack needs permissions to create SNS topics, CloudWatch alarms, EventBridge rules, and (if enabling auto-remediation) Lambda functions and IAM roles. The CAPABILITY_IAM and CAPABILITY_NAMED_IAM capabilities must be acknowledged when creating the stack.
• Amazon S3 bucket security - The CloudTrail log Amazon S3 bucket must have Block Public Access enabled (all four settings), server-side encryption (SSE-S3 or SSE-AWS KMS), versioning enabled, and access logging configured. For production environments, enable Multi-Factor Authentication (MFA) Delete on the bucket.
• AWS CLI (optional) - For command-line deployment and the clean-up step.

[!WARNING] Template Does Not Validate Amazon S3 Bucket Security Settings The AWS CloudFormation template does not validate Amazon S3 bucket security settings. The stack deploys successfully even if prerequisites are not met, leaving a potential gap in your CloudTrail log protection. Verify all prerequisites using the AWS CLI commands before deployment.

For continuous enforcement after deployment, enable the following AWS Config managed rules to detect and alert if the bucket's security configuration degrades:

• s3-bucket-public-read-prohibited - detects if Block Public Access is disabled
• s3-bucket-server-side-encryption-enabled - detects if default encryption is removed
• s3-bucket-versioning-enabled - detects if versioning is disabled
• s3-bucket-logging-enabled - detects if access logging is removed

Implementation Priority

Deploy in this sequence to maximize security coverage from day one:

1. Verify Amazon S3 bucket prerequisites - Confirm Block Public Access, encryption, versioning, and access logging are enabled before deploying.
2. Deploy detection-only stack - Set EnableAutoRemediation=false to deploy the core detection layer first. Confirm the Amazon SNS email subscription.
3. Validate alerting - Trigger a test event and confirm alerts arrive.
4. Enable Amazon S3 protection - Provide CloudTrailLogBucketName to apply the hardening bucket policy to the CloudTrail log delivery bucket.
5. Enable auto-remediation - After validating detection, set EnableAutoRemediation=true.
6. Apply preventive controls - Attach the MonitoringStackProtectionPolicy to IAM roles. Apply a Service Control Policy (SCP) if using AWS Organizations.
7. Multi-region rollout - Use AWS CloudFormation StackSets to deploy to all regions with active trails.

Deployment

Download the Template

Download the CloudFormation template from the repository:

cloudtrail-security-monitoring.yaml

Step-by-Step Deployment Procedure

1. Sign in to the AWS CloudFormation console.
2. Choose Create stack → With new resources (standard).
3. Under Specify template, choose Upload a template file and upload cloudtrail-security-monitoring.yaml. Choose Next.
4. Enter a Stack name (for example, cloudtrail-security-monitoring).
5. Fill in the parameters (see the table below). Choose Next.
6. On the Configure stack options page, leave defaults or add tags as needed. Choose Next.
7. On the Review page, scroll to the Capabilities section and check I acknowledge that AWS CloudFormation might create IAM resources with custom names. Choose Submit.
8. Wait for the stack status to reach CREATE_COMPLETE.
9. Check your email inbox for the SNS subscription confirmation message and click the confirmation link.

Parameters Reference

Parameter	Default	Description
CloudTrailLogGroupName	required	Name of the CloudWatch Logs log group that receives CloudTrail events. Must already exist.
NotificationEmail	required	Email address for SNS alert notifications. A confirmation email is sent after stack creation.
TrailName	required	Name of the CloudTrail trail to protect. Used to scope the auto-remediation IAM policy.
EnableAutoRemediation	false	Set to true to deploy the Lambda function that automatically re-enables logging when StopLogging is detected.

[!TIP] EnableAutoRemediation Setting EnableAutoRemediation to true deploys a Lambda function that calls cloudtrail:StartLogging automatically when a StopLogging event fires. This minimizes the window during which logging is inactive. Before enabling this feature, consider the security implications: the Lambda function has permission to call StartLogging on the specified trail. Verify that the trail name parameter is set to the correct trail and that your change management process accounts for automated changes to trail state. Review the Lambda function logs in CloudWatch Logs (/aws/lambda/CloudTrailAutoRemediation-<stack-name>) after any StopLogging event to confirm the remediation succeeded.

How This Solution Works Alongside AWS Config

AWS Config and this near real-time detection solution are complementary layers in a defense-in-depth strategy, each bringing distinct strengths that reinforce the other.

AWS Config managed rules provide continuous compliance evaluation of your CloudTrail configuration. Three managed rules are particularly relevant:

• cloudtrail-enabled - Verifies that at least one CloudTrail trail is active in the account.
• cloud-trail-log-file-validation-enabled - Checks that log file validation is turned on, ensuring log integrity.
• cloudtrail-s3-dataevents-enabled - Confirms that Amazon S3 data event logging is configured.

AWS Config evaluates these rules on a schedule and whenever a configuration change is recorded. It maintains a full configuration history and can trigger AWS Systems Manager Automation runbooks for remediation. This makes it an excellent tool for drift detection over time, compliance reporting, and understanding the history of configuration changes.

This solution adds a near real-time operational detection layer on top of that foundation. Where AWS Config evaluates compliance periodically, EventBridge rules match CloudTrail management events shortly after the API call is made. Where AWS Config records that a trail was deleted, this solution fires an SNS alert to your security team and if auto-remediation is enabled, calls StartLogging before the next AWS Config evaluation cycle even runs.

Together, the two layers give you both the historical compliance record that AWS Config excels at and the immediate operational response that this solution provides.

[!TIP] Defense-in-Depth: Neither layer replaces the other. AWS Config gives you compliance history and drift detection; this solution gives you near real-time alerting and automated response. Deploying both maximizes your security posture.

Detection Architecture

The solution uses two parallel detection paths that both route to the same SNS topic, so your security team receives alerts from whichever layer fires first.

Figure 1: CloudTrail Security Monitoring detection architecture

CloudWatch Metric Filter and Alarm Layer

CloudTrail delivers management events to a CloudWatch Logs log group. Three metric filters parse those log entries and emit a count metric whenever a matching API call appears:

Event	Metric	Namespace
StopLogging	StopLoggingCount	CloudTrailSecurityMonitoring
DeleteTrail	DeleteTrailCount	CloudTrailSecurityMonitoring
UpdateTrail	UpdateTrailCount	CloudTrailSecurityMonitoring

A CloudWatch alarm watches each metric with a one-minute evaluation period and a threshold of zero. Any count greater than zero transitions the alarm to ALARM state and publishes a notification to the SNS topic. This layer provides durable metric history that you can visualize on CloudWatch dashboards and use for trending over time.

EventBridge Rule Layer

EventBridge matches CloudTrail management events directly on the default event bus shortly after the API call is made, before the log entry has even been delivered to CloudWatch Logs. Two EventBridge rules are deployed:

Trail tampering rule - matches StopLogging, DeleteTrail, and UpdateTrail events from aws.cloudtrail and routes them to the SNS topic.

Amazon S3 bucket protection rule - matches DeleteBucket, DeleteBucketPolicy, and PutBucketAcl events from aws.s3 and routes them to the SNS topic. This protects the Amazon S3 bucket that receives your CloudTrail log files from destructive operations.

SNS Alert Delivery

Both detection layers route to a single SNS topic. An email subscription delivers alerts to your security team. You can add additional subscriptions (SMS, Lambda, SQS, HTTP endpoint) directly to the topic after deployment.

The topic policy restricts publishing to the owning account, cloudwatch.amazonaws.com, and events.amazonaws.com, each scoped to the account via aws:SourceAccount. This prevents unintended cross-account publishing.

Optional Lambda Auto-Remediation

When EnableAutoRemediation is set to true, the stack also deploys a Lambda function that automatically calls cloudtrail:StartLogging when a StopLogging event fires. The function:

1. Extracts the trail Amazon Resource Name (ARN) from event.detail.requestParameters.name.
2. Calls cloudtrail:StartLogging with that ARN.
3. Logs the trail ARN and a success timestamp to CloudWatch Logs.
4. On any error, logs the full exception details and re-raises so EventBridge records the invocation as failed.

The IAM execution role uses least-privilege permissions: cloudtrail:StartLogging is scoped to the specific trail ARN, and logging permissions are scoped to the Lambda log group.

Multi-Region Deployment

By default, CloudTrail creates an account-level trail that captures management events only in the region where it is configured. An account-level trail delivers logs to CloudWatch Logs only in that same region. This means EventBridge rules and CloudWatch metric filters in one region cannot detect unauthorized configuration changes that occur in another region. To protect all regions where you have active trails, you must deploy this stack in each region.

Per-region requirements:

• Each region must have a CloudTrail trail configured to deliver logs to a CloudWatch Logs log group. The CloudTrailLogGroupName parameter must reference the log group in the same region as the stack.
• EventBridge rules are regional - a rule in us-east-1 only matches events in us-east-1.
• The SNS topic and Lambda function (if enabled) are also regional resources.

[!TIP] Account-Level Trail Limitation: If you are using a default account-level trail, it only delivers to CloudWatch Logs in the trail's home region. To get cross-region coverage, you must either configure a multi-region trail (which captures events from all regions but still delivers to CloudWatch Logs in a single region) or deploy individual trails in each region. For this solution to work in a given region, a CloudWatch Logs log group receiving CloudTrail events must exist in that region.

AWS Organizations trails:

If you use an AWS Organizations trail, a single trail can cover all accounts in your organization. However, this solution must still be deployed in each region where you want near real-time alerting. The organizational trail delivers logs to a centralized Amazon S3 bucket, but the CloudWatch Logs log group and EventBridge rules remain regional.

Scaling with AWS CloudFormation StackSets:

For multi-region or multi-account deployments, use AWS CloudFormation StackSets to deploy the stack consistently across all target regions and accounts. StackSets let you specify a list of target regions and accounts, and AWS CloudFormation handles the parallel deployment. Use service-managed StackSets with AWS Organizations integration to automatically include new accounts as they join your organization.

aws cloudformation create-stack-set \
  --stack-set-name cloudtrail-security-monitoring \
  --template-url https://s3.amazonaws.com/your-bucket/cloudtrail-security-monitoring.yaml \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \
  --parameters \
    ParameterKey=CloudTrailLogGroupName,ParameterValue=aws-cloudtrail-logs \
    ParameterKey=NotificationEmail,ParameterValue=security@example.com \
    ParameterKey=TrailName,ParameterValue=my-trail

Incident Response Playbook

When an alert fires, follow these steps:

1. Confirm the alert is genuine

Check the SNS notification for the event source, event name, and the principal that made the API call (detail.userIdentity). Determine whether the action was authorized (for example, a planned maintenance window or an infrastructure-as-code deployment).

2. Assess the impact

• For StopLogging: Determine how long logging was stopped. Check CloudTrail for the most recent events before the gap. If auto-remediation is enabled, verify that StartLogging was called successfully by checking the Lambda function logs in CloudWatch Logs (/aws/lambda/CloudTrailAutoRemediation-<stack-name>).
• For DeleteTrail: The trail cannot be automatically restored. Create a new trail immediately and configure it to deliver logs to the same CloudWatch Logs log group and Amazon S3 bucket.
• For UpdateTrail: Review the change. Common concerns include disabling log file validation, changing the Amazon S3 bucket destination, or removing CloudWatch Logs integration.
• For Amazon S3 bucket events (DeleteBucket, DeleteBucketPolicy, PutBucketAcl): Assess whether log delivery has been disrupted. Restore the bucket policy if needed and verify that CloudTrail can still deliver logs.

3. Contain and remediate

• Revoke the credentials of the principal that made the unauthorized change if the action was not authorized.
• Re-enable logging if it was stopped and auto-remediation did not fire or failed.
• Restore the trail configuration to its expected state using your infrastructure-as-code baseline.
• Review IAM policies to understand how the principal obtained the permissions to make the change.

4. Document and review

• Record the incident timeline, the principal involved, and the remediation steps taken.
• Review CloudTrail events for the 24 hours before the unauthorized configuration change to identify any reconnaissance activity.
• Consider adding an AWS Config rule or SCP to prevent StopLogging and DeleteTrail calls from non-administrative principals. See the Strengthening Trail Deletion Prevention section for detailed guidance.

Security Responsibilities

This solution follows the AWS Shared Responsibility Model. Understanding the division of responsibilities helps you deploy and operate the solution securely.

Strengthening Trail Deletion Prevention

This solution detects and alerts on DeleteTrail events within seconds and includes the MonitoringStackProtectionPolicy IAM Managed Policy to prevent deletion of the monitoring resources themselves. However, trail deletion is the one event that cannot be automatically reversed — a deleted trail must be manually recreated. The following additional controls address this residual risk and are recommended as complementary layers.

Option 1: AWS Service Control Policy (Recommended for AWS Organizations)

If your account is part of an AWS Organization, a Service Control Policy (SCP) is the strongest available preventive control. An SCP applies to every IAM principal in the account — including administrators — and cannot be overridden by account-level IAM policies.

Apply the following SCP to the target OU or account, substituting your break-glass role ARN:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyCloudTrailDeletion",
      "Effect": "Deny",
      "Action": [
        "cloudtrail:DeleteTrail",
        "cloudtrail:StopLogging",
        "cloudtrail:UpdateTrail"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalArn": [
            "arn:aws:iam::<ACCOUNT_ID>:role/BreakGlassRole"
          ]
        }
      }
    }
  ]
}

This SCP allows only the designated break-glass role to delete or stop a trail. All other principals — including those with AdministratorAccess — are denied. For guidance on creating and attaching SCPs, see the AWS Organizations SCP documentation.

[!TIP] Break-Glass Role: The break-glass role should be tightly controlled: no standing access, MFA required for assumption, and all usage logged and alerted. This provides protection against unauthorized trail deletion while permitting legitimate emergency operations through the break-glass role.

Option 2: IAM Permission Boundary (Standalone Accounts)

For accounts not in AWS Organizations, apply an IAM permission boundary to all roles and users that denies cloudtrail:DeleteTrail. A permission boundary limits the maximum permissions an IAM entity can have, regardless of what their identity-based policies allow.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyTrailDeletion",
      "Effect": "Deny",
      "Action": "cloudtrail:DeleteTrail",
      "Resource": "*"
    }
  ]
}

Note that permission boundaries do not apply to the AWS account root user. Verify that root user access is secured with MFA and that root credentials are not used for routine operations.

Option 3: AWS Organizations Trail (Architectural Prevention)

An AWS Organizations trail is created in the management account and automatically covers all member accounts in the organization. A member account cannot delete an organizational trail — only the management account has that permission. This provides architectural prevention rather than policy-based prevention.

If you are already using AWS Organizations, consider migrating from per-account trails to a single organizational trail. This can simplify multi-account log aggregation and reduce the operational overhead of managing trails across accounts.

For guidance, see Creating a trail for an organization in the AWS CloudTrail documentation.

Option 4: AWS Config Automated Remediation

Enable the cloudtrail-enabled AWS Config managed rule and configure an AWS Systems Manager Automation remediation runbook. If a trail is deleted and not recreated within the evaluation window, AWS Config marks the account as non-compliant and can automatically trigger a runbook to recreate the trail with the correct configuration.

This provides a recovery mechanism when prevention controls are not in place or are bypassed.

[!TIP] Recommended Combination: For the strongest posture, combine Option 1 (SCP) with Option 3 (Organizations trail). The SCP prevents deletion by any principal in the member account, and the Organizations trail provides coverage so that even if a member account trail is somehow removed, the organizational trail in the management account continues to capture all activity.

Clean Up

To remove all resources created by this stack, delete the CloudFormation stack. The SNS topic is configured with DeletionPolicy: Retain and UpdateReplacePolicy: Retain, so it is preserved after stack deletion to prevent accidental loss of alert subscriptions. Delete the topic manually if it is no longer needed.

aws cloudformation delete-stack --stack-name cloudtrail-security-monitoring

To also delete the retained SNS topic:

aws sns delete-topic --topic-arn <AlertTopicArn>

The AlertTopicArn value is available in the CloudFormation stack Outputs tab.

Conclusion

This solution adds a near real-time detection and automated response layer to your CloudTrail security posture. By combining CloudWatch metric filters and alarms with EventBridge rules, you get both durable metric history and fast event-driven alerting. The optional Lambda auto-remediation function minimizes the window during which logging is inactive, and the Amazon S3 bucket protection rules extend coverage to the log delivery bucket itself.

Deployed alongside AWS Config managed rules (cloudtrail-enabled, cloud-trail-log-file-validation-enabled, cloudtrail-s3-dataevents-enabled), this solution forms a comprehensive defense-in-depth strategy: AWS Config provides continuous compliance evaluation and configuration history, while this stack provides the operational alerting and automated response that keeps your audit trail intact in near real time.